Skip to content

[fix](mc)Fixed the issue that maxcompute catalog can only read part of the timestamp data #49600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Mar 31, 2025
Merged

[fix](mc)Fixed the issue that maxcompute catalog can only read part of the timestamp data #49600

morningman merged 1 commit into from
Mar 31, 2025

Conversation

@hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Mar 27, 2025 •
edited
Loading

What problem does this PR solve?

Related pr : #48768

Problem Summary:
When using the maxcompute storage api to read the mc timestamp column, if the timestamp precision is required to be at the nanosecond level, only data in the range of 1970-01-01 to 2038-01-19 can be read. Otherwise, an error will be reported: ErrorCode=ODPS-0422214, ErrorMessage=ODPS-0422214: Read data failed - RuntimeTimestamp data overflow.
This pr changes the precision to milliseconds to solve this problem.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 27, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

hubgeter commented Mar 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-H: Total hot run time: 34055 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0c5cf3131fcd36db0b35e3fc1ea2e277ef91094, data reload: false

------ Round 1 ----------------------------------
q1	26268	5121	5037	5037
q2	2067	292	164	164
q3	10411	1241	696	696
q4	10217	990	541	541
q5	7541	2352	2327	2327
q6	189	162	129	129
q7	926	748	620	620
q8	9295	1260	1092	1092
q9	6895	5215	5074	5074
q10	6862	2330	1887	1887
q11	473	277	302	277
q12	357	362	234	234
q13	17774	3711	3156	3156
q14	231	225	207	207
q15	526	486	492	486
q16	622	619	589	589
q17	577	866	339	339
q18	7676	7201	7116	7116
q19	1245	956	548	548
q20	310	333	190	190
q21	3893	3318	2370	2370
q22	1045	1003	976	976
Total cold run time: 115400 ms
Total hot run time: 34055 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5157	5123	5175	5123
q2	246	333	236	236
q3	2228	2712	2422	2422
q4	1493	1859	1485	1485
q5	4557	4487	4429	4429
q6	210	164	131	131
q7	2029	1902	1761	1761
q8	2611	2594	2526	2526
q9	7325	7191	7146	7146
q10	2986	3166	2748	2748
q11	575	520	527	520
q12	689	752	602	602
q13	3506	3853	3328	3328
q14	302	306	280	280
q15	516	478	471	471
q16	675	682	643	643
q17	1139	1524	1416	1416
q18	7840	7750	7577	7577
q19	817	815	913	815
q20	1921	1963	1845	1845
q21	5311	4870	4824	4824
q22	1077	1055	1027	1027
Total cold run time: 53210 ms
Total hot run time: 51355 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-DS: Total hot run time: 193706 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0c5cf3131fcd36db0b35e3fc1ea2e277ef91094, data reload: false

query1	1408	1099	1032	1032
query2	6382	1903	1948	1903
query3	11021	4687	4331	4331
query4	54150	25653	23014	23014
query5	5170	662	476	476
query6	322	191	201	191
query7	4864	513	279	279
query8	303	266	244	244
query9	5542	2570	2573	2570
query10	406	341	263	263
query11	15221	15000	14908	14908
query12	149	107	101	101
query13	1011	502	395	395
query14	10242	6462	6560	6462
query15	201	188	178	178
query16	7102	707	508	508
query17	1090	751	608	608
query18	1575	427	330	330
query19	205	206	179	179
query20	136	134	123	123
query21	222	176	107	107
query22	4622	4712	4513	4513
query23	34177	33438	33378	33378
query24	6735	2452	2432	2432
query25	451	472	405	405
query26	671	267	149	149
query27	2146	504	343	343
query28	3114	2459	2461	2459
query29	595	581	451	451
query30	285	226	190	190
query31	877	883	833	833
query32	75	70	59	59
query33	466	403	305	305
query34	758	857	520	520
query35	815	848	754	754
query36	967	1027	897	897
query37	121	101	78	78
query38	4187	4249	4523	4249
query39	1466	1466	1459	1459
query40	238	118	107	107
query41	52	55	49	49
query42	125	109	106	106
query43	499	521	478	478
query44	1389	833	828	828
query45	186	173	170	170
query46	865	1033	663	663
query47	1858	1920	1835	1835
query48	385	423	303	303
query49	707	528	435	435
query50	713	772	434	434
query51	4281	4270	4324	4270
query52	110	110	97	97
query53	224	255	185	185
query54	493	497	428	428
query55	89	81	87	81
query56	301	273	262	262
query57	1189	1190	1185	1185
query58	250	249	253	249
query59	2771	2781	2697	2697
query60	304	293	268	268
query61	140	132	131	131
query62	747	763	667	667
query63	225	193	193	193
query64	1639	1116	697	697
query65	4409	4265	4310	4265
query66	721	412	305	305
query67	15931	15875	15585	15585
query68	6143	844	524	524
query69	544	342	262	262
query70	1189	1130	1134	1130
query71	436	305	269	269
query72	5825	4805	4958	4805
query73	1313	694	348	348
query74	8903	9279	8905	8905
query75	3267	3226	2761	2761
query76	3843	1200	755	755
query77	522	375	272	272
query78	10034	10215	9415	9415
query79	1506	817	578	578
query80	634	615	445	445
query81	494	251	222	222
query82	212	125	102	102
query83	176	168	160	160
query84	293	92	75	75
query85	742	352	313	313
query86	353	288	273	273
query87	4496	4556	4433	4433
query88	2828	2241	2220	2220
query89	397	304	277	277
query90	1648	203	213	203
query91	140	142	110	110
query92	73	62	58	58
query93	1739	1086	587	587
query94	681	412	310	310
query95	340	275	262	262
query96	484	577	365	365
query97	3141	3239	3118	3118
query98	221	211	207	207
query99	1338	1389	1293	1293
Total cold run time: 294046 ms
Total hot run time: 193706 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

ClickBench: Total hot run time: 31.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0c5cf3131fcd36db0b35e3fc1ea2e277ef91094, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.11	0.11
query3	0.25	0.20	0.19
query4	1.59	0.20	0.20
query5	0.59	0.59	0.58
query6	1.18	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.58	0.52	0.55
query10	0.57	0.59	0.58
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.70	2.72	2.66
query15	0.94	0.86	0.85
query16	0.39	0.38	0.38
query17	1.03	1.02	1.00
query18	0.21	0.21	0.20
query19	1.83	2.00	1.82
query20	0.01	0.01	0.01
query21	15.35	0.88	0.55
query22	0.75	1.17	0.67
query23	14.93	1.39	0.65
query24	6.86	1.60	1.50
query25	0.51	0.20	0.08
query26	0.55	0.15	0.13
query27	0.05	0.05	0.06
query28	10.91	0.84	0.43
query29	12.52	3.98	3.31
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.24	0.55	0.47
query33	3.07	3.07	3.03
query34	15.71	5.09	4.53
query35	4.54	4.62	4.47
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.25 s
Total hot run time: 31.83 s

morningman
morningman approved these changes Mar 31, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 31, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 31, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 31, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 58b12f6 into apache:master Mar 31, 2025
28 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Mar 31, 2025
@hubgeter
[fix](mc)Fixed the issue that maxcompute catalog can only read part o…
a0d8beb 
…f the timestamp data (#49600)

### What problem does this PR solve?

Related pr : #48768

Problem Summary:
When using the maxcompute storage api to read the mc timestamp column,
if the timestamp precision is required to be at the nanosecond level,
only data in the range of 1970-01-01 to 2038-01-19 can be read.
Otherwise, an error will be reported: `ErrorCode=ODPS-0422214,
ErrorMessage=ODPS-0422214: Read data failed - RuntimeTimestamp data
overflow`.
This pr changes the precision to milliseconds to solve this problem.
@github-actions github-actions bot mentioned this pull request Mar 31, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Mar 31, 2025
@hubgeter
[fix](mc)Fixed the issue that maxcompute catalog can only read part o…
9cfad0f 
…f the timestamp data (#49600)

### What problem does this PR solve?

Related pr : #48768

Problem Summary:
When using the maxcompute storage api to read the mc timestamp column,
if the timestamp precision is required to be at the nanosecond level,
only data in the range of 1970-01-01 to 2038-01-19 can be read.
Otherwise, an error will be reported: `ErrorCode=ODPS-0422214,
ErrorMessage=ODPS-0422214: Read data failed - RuntimeTimestamp data
overflow`.
This pr changes the precision to milliseconds to solve this problem.
@github-actions github-actions bot mentioned this pull request Mar 31, 2025
Merged
yiguolei pushed a commit that referenced this pull request Apr 1, 2025
@github-actions @hubgeter
branch-2.1: [fix](mc)Fixed the issue that maxcompute catalog can only…
e898dbb 
… read part of the timestamp data #49600 (#49706)

Cherry-picked from #49600

Co-authored-by: daidai <changyuwei@selectdb.com>
dataroaring pushed a commit that referenced this pull request Apr 22, 2025
@github-actions @hubgeter
branch-3.0: [fix](mc)Fixed the issue that maxcompute catalog can only…
a8fd878 
… read part of the timestamp data #49600 (#49705)

Cherry-picked from #49600

Co-authored-by: daidai <changyuwei@selectdb.com>
@yiguolei yiguolei mentioned this pull request May 13, 2025
Closed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@hubgeter @koarz
[fix](mc)Fixed the issue that maxcompute catalog can only read part o…
bef06b7 
…f the timestamp data (apache#49600)

### What problem does this PR solve?

Related pr : apache#48768

Problem Summary:
When using the maxcompute storage api to read the mc timestamp column,
if the timestamp precision is required to be at the nanosecond level,
only data in the range of 1970-01-01 to 2038-01-19 can be read.
Otherwise, an error will be reported: `ErrorCode=ODPS-0422214,
ErrorMessage=ODPS-0422214: Read data failed - RuntimeTimestamp data
overflow`.
This pr changes the precision to milliseconds to solve this problem.
@gavinchou gavinchou mentioned this pull request Jun 11, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.6-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@hubgeter @hello-stephen @doris-robot @morningman @CalvinKirs @yiguolei @dataroaring